Papers with linguistic understanding
Exploring Inherent Biases in LLMs within Korean Social Context: A Comparative Analysis of ChatGPT and GPT-4 (2024.naacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been criticized for perpetuating stereotypes against diverse groups based on race, sexual orientation, and other attributes. |
| Approach: | They devised a set of prompts that reflect major societal issues in Korea and assign varied personas to both ChatGPT and GPT-4 to assess the toxicity of the generated sentences. |
| Outcome: | The proposed model produces twice the level of toxic content as ChatGPT and GPT-4 under certain conditions. |
VLIS: Unimodal Language Models Guide Multimodal Language Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models face challenges in tasks that require complex linguistic understanding. |
| Approach: | They propose a framework that combines visual conditioning and linguistic understanding of unimodal text-only language models without further training to improve vision-language models. |
| Outcome: | The proposed framework improves vision-language models on diverse tasks including commonsense understanding and complex text generation. |
Contextualized Usage-Based Material Selection (L18-1)
Copied to clipboard
| Challenge: | Currently, authentic linguistic examples for a given keyword search are organized alphabetically according to context. |
| Approach: | They propose to use NLP-functionalities to organize usage-based examples from corpora . they group retrieved examples on syntactic grounds, then show semantic similarity within phrasal slots . |
| Outcome: | The proposed system would help language learners and other end users to benefit from a distributional linguistic analysis. |
Probing the Depths of Language Models’ Contact-Center Knowledge for Quality Assurance (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in large Language Models (LMs) have significantly enhanced their capabilities across various domains, including natural language understanding and domain knowledge. |
| Approach: | They propose methods to transfer domain-specific knowledge to smaller models by leveraging evaluation plans generated by more knowledgeable models with optional human-in-the-loop refinement to enhance the capabilities of smaller models. |
| Outcome: | The proposed models improve 18.95% on an in-house QA dataset on a contact-center quality assurance task. |
Evaluating Lexical Proficiency in Neural Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Natural Language Processing have been significantly shaped by the Deep Learning tsunami and the introduction of Transformer-based Language Models. |
| Approach: | They validated a framework to assess the lexical proficiency and linguistic creativity of Transformer-based Language Models (LMs) by analyzing performance of LMs of different sizes across tasks involving the generation, definition, and contextual usage of lexicals, neologisms, and nonce words. |
| Outcome: | The framework evaluates LMs in mono- and multilingual configuration across tasks involving the generation, definition, and contextual usage of lexicalized words, neologisms, and nonce words. |
MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning (2026.findings-eacl)
Copied to clipboard
Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin, Md Mofijul Islam, Swakkhar Shatabda
| Challenge: | Existing benchmarks primarily focus on English or a narrow subset of high-resource languages, leaving significant gaps in assessing multilingual and cross-lingual mathematical reasoning. |
| Approach: | They propose a parallel multilingual benchmark for mathematical problem solving and reasoning that encompasses 2,890 parallel Bangla-English gold standard artifacts. |
| Outcome: | The proposed model encompasses 2,890 parallel Bangla-English gold standard artifacts, totaling 30K aligned question–answer pairs across thirteen languages, representing high-, medium-, and low-resource linguistic settings. |
COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve compositional reasoning in vision language models are resource-intensive or do not provide an interpretable reasoning process. |
| Approach: | They propose a method that augments VLM outputs with carefully designed neurosymbolic concept trees learned from LLMs to improve VLM’s linguistic reasoning. |
| Outcome: | Empirical results show that COCO-Tree significantly improves compositional generalization and provides a rationale behind VLM predictions. |
Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | PIXEL is a vision transformer that has been pre-trained on rendered text . however, it is not able to outperform monolingual subwords like BERT . |
| Approach: | They propose to use PIXEL as a vision transformer to train on rendered text to explore the gap between its visual and linguistic understanding. |
| Outcome: | The proposed model outperforms monolingual subword models in most other contexts, but it lacks the linguistic knowledge to perform in language tasks. |
Visual Evidence Prompting Mitigates Hallucinations in Large Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | LVLMs have shown impressive progress by integrating visual perception with linguistic understanding to produce contextually grounded outputs. |
| Approach: | They propose a visual evidence prompting method to mitigate hallucinations in large vision-language models by using small visual models to complement them. |
| Outcome: | The proposed method reduces hallucinations by reducing false activation and enhancing correct ones. |
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (D18-1)
Copied to clipboard
| Challenge: | Existing QA datasets focus on linguistic understanding, but OpenBookQA probes deeper understanding of topic and language. |
| Approach: | They propose a dataset modeled after open book exams for question answering . the open book is a set of 1326 elementary level science facts . human performance on OpenBookQA is close to 92%, they show . |
| Outcome: | The proposed dataset is modeled after open book exams for question answering . human performance on OpenBookQA is close to 92%, but many state-of-the-art QA methods perform poorly . |
NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current research directions rely on synthetic data generated by translating English corpora, which often fails to represent the cultural heritage and values of local communities. |
| Approach: | They propose a method to create and retrieve pre-training data tailored to a specific community . they use Egyptian and Moroccan dialects as testbeds to test their understanding . |
| Outcome: | The proposed method outperforms existing Arabic-aware LLMs and performs on par with larger models. |
Careful Selection of Knowledge to Solve Open Book Question Answering (P19-1)
Copied to clipboard
| Challenge: | Open book question answering requires deeper reasoning involving linguistic understanding and common knowledge. |
| Approach: | They propose a dataset that mimics open book question answering to achieve 72.0% accuracy. |
| Outcome: | The proposed dataset achieves 72.0% accuracy, an 11.6% improvement over the current state of the art. |
Prompting-based Synthetic Data Generation for Few-Shot Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | Language models have boosted the performance of Question Answering, but data annotation is costly. |
| Approach: | They propose to use large language models to improve Question Answering performance . they argue that domain-agnostic knowledge from LMs is sufficient to create a well-curated dataset. |
| Outcome: | The proposed model outperforms state-of-the-art approaches on few-shot Question Answering. |
CrafText Benchmark: Advancing Instruction Following in Complex Multimodal Open-Ended World (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to assess instruction following in dynamic and uncertain environments are limited and limited in their ability to adapt to the world's volatility and interdependencies. |
| Approach: | They propose a benchmark for evaluating instruction following in a multimodal environment with diverse instructions and dynamic interactions. |
| Outcome: | The proposed method measures an agent’s ability to generalize to novel instruction formulations and dynamically evolving task configurations, providing a rigorous test of both linguistic understanding and adaptive decision-making. |
Can Small Vision–Language Models Perform Sign Language Translation? (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have shown strong generalization across multimodal tasks, but their capacity to handle sign language translation (SLT) remains unclear. |
| Approach: | They propose entity- and semantics-aware metrics tailored for SLT to evaluate their performance. |
| Outcome: | The proposed metrics highlight the limitations of general-purpose VLMs to SLT, unlike their applicability in other tasks. |
AraEval: An Arabic Multi-Task Evaluation Suite for Large Language Models (2025.emnlp-main)
Copied to clipboard
Alhanoof Althnian, Norah A. Alzahrani, Shaykhah Z. Alsubaie, Eman Albilali, Ahmed Abdelali, Nouf M. Alotaibi, M Saiful Bari, Yazeed Alnumay, Abdulhamed Alothaimen, Maryam Saif, Shahad D. Alzaidi, Faisal Abdulrahman Mirza, Yousef Almushayqih, Mohammed Al Saleem, Ghadah Alabduljabbar, Abdulmohsen Al-Thubaity, Areeb Alowisheq, Nora Al-Twairesh
| Challenge: | AraEval is a suite of evaluation tasks designed to assess the advanced knowledge, reasoning, truthfulness, and instruction following capabilities of large language models. |
| Approach: | They propose to use AraEval to assess the advanced knowledge, reasoning, truthfulness, and instruction following capabilities of large language models in the Arabic context. |
| Outcome: | The evaluation suite covers a broad spectrum of domains, including science, history, religion, and literature. |